Journal of the American Medical Informatics Association

Large language models (LLMs) screen titles and abstracts without review-specific training, but generating screening decisions as text takes processing time and incurs API charges. We evaluated Jev, a non-generative model returning classification probabilities, on 4527 records from two systematic reviews of bipolar disorder treatments. We used human reviewers' decisions to retain records for full-text assessment as the reference standard. In the primary analysis, we asked Jev yes-or-no questions ("Noul") about whether to retain each record. Records were retained when their retention probability was at least 50%, a cutoff specified before the final evaluation runs. We also evaluated lower retention thresholds. For comparison, we gave Jev explicit include/exclude options ("Choice") and used the same prompts with GPT-5 mini and GPT-6 Astra; both LLMs used default reasoning. In the primary analysis at 50%, sensitivity and specificity were 99.6% and 98.9% for light therapy and 93.2% and 93.5% for adjunctive pharmacotherapy. Lower cutoffs selected on these data retained all reference-positive records. Compared with assessing every record manually, these cutoffs reduced the number requiring human assessment by 96.9% for light therapy and 61.3% for adjunctive pharmacotherapy. For light therapy and adjunctive pharmacotherapy, sensitivity was 92.3% and 90.0% with Jev using explicit options at 50%, 87.2% and 90.0% with GPT-5 mini, and 92.3% and 80.8% with GPT-6 Astra. Costs for screening all 4527 records once were US$0.20 for Jev using explicit options, US$2.59 for GPT-5 mini, and US$33.45 for GPT-6 Astra. Jev's probabilities could provide graded information for reviewers assessing titles and abstracts.

A safety certificate for a clinical AI promises that no more than a stated fraction of the answers it releases carry a defect that would change care. That promise is computed from past cases a clinician marked safe, and clinicians miss things. A missed defect is not filed as a disagreement, it is filed as a safe case, so the certificate rests on a record cleaner than the ward ever was and has to be widened to compensate. Widening it correctly takes one number: how often that clinician misses a defect the model produced. We show that number cannot come from the only test a hospital can run on a working reviewer, planting a known defect and counting the catches. A planted defect is as hard to spot as whoever planted it made it, a real defect is as hard to spot as the model happened to make it, and comparing the two needs the very number being sought. No safety factor recovers it, because one wide enough for every miss rate releases nothing. The failure also runs the dangerous way, because corrections that turn noise into one rate need the mistakes to be unrelated to the case, whereas a defect is missed because it read well, so the risk on record sits below the true risk at every threshold. Across four simulated reviewers that condition never holds, and testing it on a separately drawn sample is affordable only where the answer was never in doubt: a reviewer missing 85% of defects costs 185 adjudicated cases against 147,938 for one missing 3%. Adjudicating the reviewer's own cases instead removes that test and most of that cost, 498 cases for the careful reviewer, so modelling the noise costs more than measuring it. Our library therefore refuses the widened certificate unless the condition is confirmed on adjudicated data, and fails closed otherwise.

Objective: To compare the accuracy, cost, and speed of a low-cost, non-generative structured-judgment model and four general-purpose large language models (LLMs) for extracting smoking status, pack-years, and quit date from clinical notes to support lung cancer screening (LCS) clinical decision support (CDS). Materials and Methods: We built a synthetic, shareable benchmark of 3,000 outpatient notes in three conditions: 1,000 template-generated notes, 1,000 realistic, ``messy'' notes written by an LLM from structured facts, and 1,000 ``messy'' notes that required complex arithmetic to determine pack-years and quit dates. Ground truth reference labels were programmatically generated before each note was created. We compared TypeSafe Jev 1.13 with Claude Haiku 4.5, Claude Sonnet 5, GPT-6 Luna and GPT-6 Sol, each using an identical structured output schema. We determined United States Preventive Services Task Force (USPSTF) 2021 and American Cancer Society (ACS) 2023 eligibility from each system's output in code. Results: Jev made the correct eligibility decision for 99.1%, 99.9%, and 94.4% of notes in the three conditions, compared with 97.8%, 98.4%, and 98.1% for Haiku, 100.0%, 99.6%, and 99.8% for Sonnet, 99.7%, 98.8%, and 98.5% for Luna, and 100.0%, 99.8%, and 99.6% for Sol. In the complex condition, Jev produced 27 false positive and 14 false negative screening flags per 1,000 notes, the most of any system. Luna cost $0.12 to $0.21 per 1,000 notes and Jev $0.61 to $0.64, compared with $2.44 to $4.22 for Sol, $3.37 to $4.43 for Haiku, and $3.76 to $6.61 for Sonnet. Jev was the fastest system (median 0.45 to 1.21 seconds per note, compared with 1.71 to 3.50 seconds for the others). Discussion: All models tested had high accuracy. The structured-judgment model had the lowest latency, which could allow for synchronous use in clinical decision support systems rather than batched, cached invocation. The cost of the structured-judgment model was low, but the newest cost-efficient LLM, Luna, was less expensive and more accurate on the complex notes, though its latency was higher. Accuracy, latency, and cost are all moving quickly as new models are released. Conclusion: Fast, low-cost language models could make real-time, note-based CDS practical, and current models are highly accurate. Prices and capabilities are changing rapidly, so continuous benchmarking and system updates are essential.

Title and abstract screening is among the most labor-intensive steps of systematic reviews. Jev, a low-cost AI decision model, scores records against eligibility criteria without screening labels. We scored 169,982 records from 128 reviews and, after tuning on development reviews, evaluated 23 held-out reviews once and 31 Cochrane reviews post hoc. Jev outranked the best free ranker in all 23 held-out reviews (mean AUC 0.966 vs 0.831) and came within 0.011 AUC of Claude Opus 5.5 on 1999 records at about 1/700 of its list-price cost (US$0.02/1000 records). Combined with active learning, it raised work saved at 95% recall from 0.659 to 0.806. A fixed score threshold fell below 95% recall in 3 of 28 Cochrane reviews, whereas Jev ranking with statistical stopping (post hoc) reached 95% recall in all 51 reviews, reading 86-88% of records. Label-free ranking with Jev was accurate and inexpensive, but stopping safely required screening decisions.

Background. Chest radiographs (CXRs) are the most common imaging exam performed, but CXR reports can be nonspecific due to incomplete history, relying on vague terms like opacities to convey diagnostic uncertainty. We prospectively evaluated whether introducing a clinical dashboard with relevant patient information affects CXR report specificity and efficiency, report concordance with discharge diagnoses, and follow-up chest CT utilization. Methods. In this stepped-wedge cluster randomized trial (NCT06785246), a clinical dashboard for radiologists was rolled out with user education sessions across 15 service areas of an integrated health system from June 2024 to June 2025. Emergency department (ED) and inpatient CXRs were included. A large language model (GPT-4o) was validated using a Delphi panel of three experts to classify report specificity and CXR report diagnosis. A subsequent model release (GPT-5.4) graded each CXR report impression as negative, specific abnormal, or nonspecific abnormal. It also judged whether an abnormal impression matched an ED discharge diagnosis. The primary outcome was the proportion of specific reports (negative or specific abnormal). Secondary outcomes were report-discharge diagnosis concordance, follow-up chest CT, antibiotic and diuretic orders, and CXR reporting time. Mixed-effects models adjusted for time, patient age, and radiograph view, with random effects for radiologists, facilities, and service area. Results. The trial included 404,860 ED and inpatient encounter STAT priority CXRs read by 387 radiologists (256,303 control; 148,557 intervention). The adjusted proportion of specific reports was 82.2% in control periods and 82.1% in intervention periods (difference, -0.10 percentage points; 95% CI, -0.53 to 0.34; P=.66). Among abnormal reports, concordance with a discharge diagnosis rose from 40.9% to 42.2% (difference, 1.24 points; 95% CI, 0.20 to 2.28; P=.02). Follow-up chest CT fell from 11.4% to 10.9% (difference, -0.55 points; 95% CI, -0.92 to -0.17; P=.004). Antibiotic orders (24.0% vs 23.9%) and diuretic orders (14.7% vs 14.8%) within 7 days did not change, nor did reporting time (difference, -0.8 seconds; 95% CI, -2.7 to 1.0). Conclusions. Giving radiologists EHR context through a dashboard did not change the specificity of STAT priority CXR reports or prescribing patterns. It was associated with small improvements in diagnostic concordance and follow-up CT use, without slowing reporting. Validated large language models make it feasible to measure report quality across a health system.

Objective: To estimate Long COVID undercoding in electronic health records (EHRs) against federal survey prevalence, and to evaluate a predictor of later coding that anyone can rebuild from counts a research enclave releases. Materials and Methods: Among 23,790 All of Us adults with COVID-19, we compared coded shares with Household Pulse Survey prevalence. Among 23,300 adults uncoded at day 90, we predicted a first U09.9 code by day 455. We fitted naive Bayes and linear discriminant models outside the enclave from released counts and compared them with an elastic net fitted inside to the same patients and features. Tests were a random 20% of patients (4,680, 50 coded) and 18 held-out EHR sites (5,760, 60 coded). Results: Under survey-comparability assumptions, about 1 in 9 adults who ever had Long COVID carried its code, and the women-to-men coding ratio was 0.79 (95% sensitivity interval, 0.65-0.95). The released models' areas under the receiver operating characteristic curve (AUROCs) were 0.68 (95% CI, 0.60-0.75) on the patient test and 0.64 (0.55-0.72) at held-out sites, 0.049 (0.001-0.104) above the elastic net. On the patient test, the elastic net had a higher area under the precision-recall curve (difference 0.062, 0.003-0.122). Released probabilities overestimated coding risk at held-out sites. Discussion: Fitting from released counts did not consistently lower discrimination. Coded outcomes inform disease discrimination only under stated coding assumptions. Conclusion: Long COVID is substantially and unevenly undercoded, and a model rebuilt from public counts predicts its later coding with moderate discrimination.

Clinicians routinely make decisions for patients whose comorbidities, prior therapies, laboratory abnormalities, or demographic characteristics are not reflected in the supporting evidence, even though these factors can influence treatment selection. Given this gap in the evidence base, we sought to evaluate how AI models respond to clinical questions in which patient characteristics or the treatment comparison and outcome of interest may affect the applicability of available evidence. We developed and publicly released Precision Evidence Bench, a benchmark of 209 synthetic clinical questions with specified population, intervention, comparator, and outcome (PICO) elements. Each answer is assessed using a five item rubric that evaluates whether a single cited source matches the PICO and whether the answer faithfully reports its sources. The item verdicts determine one of five ratings, ranging from Precision match to Unsupported claim, without requiring a reference answer. We evaluated four frontier large language models (LLMs) using web search across all 209 questions, then repeated the evaluation with additional access to a corpus of precision evidence-based findings (pEBFs) designed to address these evidence gaps, using two corpus sizes. For every model, Precision match rates increased with the amount of precision evidence available, from 13 to 18% with web search alone to 17 to 28% with 100 million pEBFs and 41 to 56% with 500 million pEBFs. These findings suggest that access to matching evidence is an important determinant of LLM performance on precision medicine questions.

Importance: The Medicare Physician Fee Schedule values ambulatory services using survey-based estimates of physician time, a costly process vulnerable to inaccuracy. EHR audit logs are passively and universally collected and likely meet the American Medical Association Relative Value Scale Update Committee (AMA RUC) Extant Database Standards for alternative data to inform service valuation estimates. However, audit logs' validity for estimating face-to-face time, a core component of many of the most used procedural codes, is unknown. Objective: To assess whether EHR audit logs can facilitate accurate estimation of physician-patient face-to-face time during ambulatory visits. Design: Single-site validation study pairing EHR audit logs with direct audio/video recordings of ambulatory primary care visits (reference standard). We assessed agreement between audit-log estimates and recorded visit duration with Bland-Altman analysis and mean absolute error. Setting: University of Pennsylvania Observer data repository, a multi-modal database of ambulatory visits to Penn Medicine primary care clinics including video and audio recordings, transcripts, clinical EHR data, and EHR audit-log data. Participants: 75 visits from 12 physicians with data collected between August 2023 and February 2025. Exposures: A rule-based algorithm estimated face-to-face time via clusters of physician EHR activity relative to patient check-in time. Inactivity thresholds were applied to differentiate intraservice time with no EHR activity (e.g., a physical exam) from the end of a visit, with longer thresholds used for longer visits. Main Outcomes and Measures: Duration of recorded face-to-face visit time. Results: Recorded face-to-face time averaged 31.6 minutes (SD, 13.9). The audit-log estimator was unbiased (mean bias: -0.63 minutes; 95% CI: -6.0 to 2.6; mean absolute error: 9.0 minutes) and estimated face-to-face time within approximately 2% (38 seconds) of the recorded visit time with no proportional bias (slope: 0.13; P = .25). The 95% limits of agreement were -25.7 to 24.4 minutes, and 37% of estimates fell within 10% of recorded duration. Conclusions and Relevance: In this validation study, EHR audit logs recovered unbiased estimates of ambulatory face-to-face time. Larger-scale studies are required to verify generalizability, but EHR audit logs offer a scalable means to improve the rigor of retrospective physician work relative value unit estimation. This study represents a first step in a potential paradigm shift for measuring components of the fee schedule across procedures, settings, and provider types.

Background Large language models (LLMs) show promise in scientific literature reviews, but cloud-based proprietary models like GPT-4 raise concerns around privacy, copyright, and licensed article access. Using a scoping review on AI applications for treatment effect estimation as a use case, this study evaluated local open-source LLMs for automated article screening in scientific literature reviews and developed a practical workflow for their implementation. Methods We extracted a sample dataset of 300 records from an ongoing scoping review on machine learning methods for treatment effect estimation using real-world medical data. We divided the dataset into training (n = 200) and testing (n = 100) datasets. We evaluated 11 locally deployed open-source LLMs from five model families (GPT-OSS, Gemma, Qwen, Mistral, Llama) for both abstract screening and full-text screening. Full-text screening was done using text converted by two PDF-to-text conversion tools: Docling and PyMuPDF4LLM. Performance was assessed using raw accuracy with 95% Wilson's confidence interval, relaxed accuracy, Cohen's kappa, and runtime. During workflow development, prompt engineering, output extraction strategies, and generation settings were iteratively optimized using the training dataset. Results GPT-OSS-20B achieved the highest raw accuracy for title and abstract screening (0.890 on the training set and 0.850 on the test set), followed closely by Gemma-4-31B-it (0.875 on the training set and 0.830 on the test set) and Gemma-4-26B-A4B-it (0.810 on the training set and 0.790 on the test set). In full-text screening, slightly higher raw accuracy and reduced false-positive rates were observed, with GPT-OSS-20B and Gemma-4 variants consistently demonstrating strong performance. Only minor variations were found between the two PDF-to-text conversion methods evaluated, though runtime varied substantially across models. Based on these findings, we compiled a set of practical recommendations for local LLM-assisted article screening. Conclusions Local LLMs can effectively support literature screening when integrated into a carefully designed workflow. Recent generations of open-source local models achieved performance comparable to human reviewers and proprietary LLMs. Local deployment also provides advantages in data privacy, copyright compliance, customization, and cost-efficiency. While human oversight remains essential, the proposed workflow demonstrated that local LLMs represent a practical and reproducible approach for accelerating evidence synthesis.

The Elixhauser comorbidity measures are among the most widely used risk adjusters in administrative health data. Their reference implementation, published by the Agency for Healthcare Research and Quality (AHRQ) as SAS programs, has eleven annual releases in two families, one screening pre-existing conditions on the present-on-admission (POA) indicator and the earlier one on Medicare Severity Diagnosis-Related Group, and no implementation of every release existed outside SAS. We developed `ecsr10`, an open-source R package covering all eleven releases of both families, with a browser-based application over the same functions, and validated it against AHRQ's own SAS programs over 40,027,114 value-level comparisons with zero disagreements. Two existing open-source reimplementations scored on the same dataset showed reproducible defects. We then used the package to quantify two choices AHRQ's software leaves to the analyst and studies seldom report, the release and the handling of diagnoses not present on admission, on the Texas Inpatient Public Use Data File, 2016 Q1-2019 Q4 (8,585,244 adult discharges from 726 hospitals), scoring one predefined cohort under every release and pre-existing-condition setting. POA handling dominated: disabling POA changed the comorbidity profile of 56.7% of admissions, and a POA-naive mortality index scored higher on discrimination (area under the curve 0.807 versus 0.789; difference 0.0183, 95% confidence interval 0.0161-0.0204) by counting in-hospital complications as pre-existing disease. POA reporting was bimodal across hospitals, so pooled hospital comparisons on such a file partly compare documentation practice. Because the cohort predates every release compared, the release contrast is a lower bound: the release chosen between v2022.1 and v2026.1 changed at least one comorbidity flag for 0.0036% of admissions, yet a single revision of ten mortality weights changed the comorbidity index for 42.4% of them. An exactly validated implementation makes such choices measurable; release, POA handling and the implementation used should be reported as study characteristics.

Reliable extraction of trial characteristics and outcomes is essential for systematic reviews, yet incorrect values can alter evidence synthesis and are difficult to detect without clear source evidence. We evaluate EviSearch, a clinical evidence-curation system that links extracted values to source passages, independently verifies cited evidence, and prioritizes higher-risk cells for review. Using ten trial publications, a 133-column schema, reference annotations, and consistent scoring across model families, we assess answer quality, inter-reading agreement, evidence reproduction, targeted review, and evidence modality. In the Qwen evaluation, EviSearch achieved 91.70% accuracy, cited 100.0% of admitted values, and reproduced 98.9% independently. Reviewing 15.6% of cells increased accuracy to 95.22%.

Objective: We explored the implementation considerations of machine learning (ML) suicide risk identification in a health system that serves primarily American Indian patients in the Southwestern United States. Previous work developed and reported the performance of suicide risk models, but traditional performance metrics do not directly reflect clinical workflows. Materials and Methods: Using EHR data collected for Emergency Department (ED) visits between 10/01/2019 and 10/02/2021, we conducted Decision Curve Analysis (DCA) to identify the standardized net benefit (SNB) of all models previously evaluated and plotted the potential workload implications of the model with the greatest SNB. Finally, we evaluated suicide screening practices during the study period to inform implementation barriers. Results: Among 9,244 patients, 139 (1.5%) had a suicide attempt or death within 90 days of an included visit. The population-specific logistic regression model had the greatest SNB compared to other approaches. When operationalized, the 95th percentile cutoff did not substantially increase clinical workloads, while the 75th percentile cutoff substantially impacted workloads. The secondary analysis on suicide screening depicted variations in screening administration over time. Discussion: While DCA identified advantages of ML suicide risk identification over current standards, the complexities of suicide-related care limited its clinical interpretability. A stepped-care framework may mitigate potential workload burdens. Training initiatives and dynamic implementation strategies will support high-quality implementation of ML suicide risk identification. Conclusion: Our analyses support the feasibility and utility of augmenting existing suicide screening practices with ML suicide risk identification, and demonstrate the importance of nuanced analyses to inform implementation considerations.

Electronic health records provide a source of real-world information for disease risk prediction, but many machine learning models learn implicit associations that are difficult to inspect or use for clinical reasoning. Causal Bayesian networks (CBNs) offer a form of causal-interpretable machine learning by representing conditional dependencies, putative directional relations, and probabilistic evidence propagation in a directed acyclic graph. However, conventional CBN structure learning becomes unstable and computationally expensive when applied to large-scale EHR data. To address this limitation, we propose UPEBNL, a scalable framework for causal-interpretable cancer prescreening based on parallel CBN learning. UPEBNL integrates adaptive data slicing, quality-aware structure aggregation, and global DAG construction to learn stable and interpretable dependency structures from large-scale observational data. We evaluated UPEBNL in high-dimensional and multi-million-sample simulations and applied it to EHR-based prescreening for esophageal and colorectal cancer. In simulations, UPEBNL improved structural recovery accuracy by nearly 40% and achieved up to a 221.28-fold speedup over conventional CBN learning strategies. For cancer risk prediction, the learned CBNs provided interpretable evidence paths and achieved validation AUCs of 0.8171 for esophageal cancer and 0.784 for colorectal cancer. Calibration and decision curve analyses further supported the reliability and clinical utility of the models. These findings suggest that scalable CBN learning can support interpretable cancer prescreening from large-scale EHR data.

Background: Many radiology decisions, such as whether a report describes a finding or how urgently a study should be read, are choices among a fixed set of answers. Decision models return a probability for each permitted answer in one step, without generating text, and an open-weight model can run within a hospital so that patient data stay local. No decision model has yet been specialized for radiology and compared with large language models (LLMs). Methods: We fine-tuned the general-purpose decision models Kev-27B (Qwen3.8-27B backbone) and Kev-9B on 36,109 records from public radiology datasets to obtain RadKev-27B and RadKev-9B, and compared them with the Kev models, the hosted decision models Jev and OpenAI Decisions, and the LLM Qwen3.8-27B, on 14,142 held-out questions in fifteen tasks (report reading, case diagnosis and classification, radiological knowledge). The prespecified primary outcome was task-averaged accuracy relative to Kev-27B. Results: RadKev-27B exceeded Kev-27B by 4.9 percentage points (95% CI 3.8 to 6.2), with the largest gain in report error detection (22.6 points); RadKev-9B exceeded Kev-9B by 6.9 points (5.5 to 8.4) and did not differ detectably from Kev-27B, a model three times its size. RadKev-27B was more accurate than the hosted decision models Jev (3.0 points, 1.7 to 4.4) and OpenAI Decisions (4.8, 3.2 to 6.4), than Qwen3.8-27B without reasoning (4.7, 3.4 to 6.0) and with reasoning (2.8 points, 1.0 to 4.6) on 1,675 questions, at 132-fold lower latency (128 ms vs 16.9 s per question). At an error rate of 5%, RadKev-27B could answer 90.5% of questions and Kev-27B 76.4%. Conclusions: RadKev-27B was more accurate than two hosted decision models and an LLM of the same size, with or without reasoning, at a fraction of the latency of reasoning, and can be deployed within an institution for categorical radiology decisions after validation on local data.

Background: Large language models (LLMs) can support systematic reviews, but accurate individual outputs do not establish whether the final synthesis preserves the clinical question, accounts for statistical dependence and incorporates corrections. Objective: To develop and evaluate a framework linking LLM-assisted evidence processing to a versioned, auditable release of synthesis outputs. Methods: We used LLM agents to interpret sources and extract data. Deterministic code enforced statistical rules; investigators resolved material ambiguities and authorized release. We specified ten release properties covering evidence identities, statistical contributions and propagation of corrections. We retrospectively evaluated six integrity domains and historical failure events in one registered prognostic review, without an external comparator or held-out domain. Results: Fifty distinct root-cause events were documented, including 12 that had changed a pooled result before correction. Forty-six were resolved, and four remained disclosed limitations. The corpus comprised 454 reports, 445 studies, 441 cohort entities and 421 dependence clusters. Forty-one of 49 registered analyses were fitted, and eight retained explicit non-fitted states. All 39 source records across five principal analysis families reached a terminal source state. Two implementations within the project agreed across 1,217 numerical comparisons. All 94 file comparisons between release and publication packages were byte-identical. Two reviewers confirmed 39 principal records after seeing the same recommendations. Conclusions: This case provides a framework for inspecting synthesized evidence together with its provenance, statistical meaning and correction history. Comparative validity, generalizability and benefit in patient-centred care require independent evaluation.

Background. Research data enclaves such as the All of Us Research Program do not allow participant-level data, or models fitted to them, to leave, which makes it hard to deliver case-finding models, which flag patients with possibly uncoded conditions for clinician review, to patients and clinicians. Objectives. We describe and technically evaluate a system that fits models outside an enclave from released summary counts and delivers them through standard health information technology interfaces. Methods. We exported dissemination-compliant summary counts from the All of Us Researcher Workbench and, outside the enclave, fitted versioned models for Long COVID, chronic fatigue, fibromyalgia, endometriosis, Lyme disease, lupus, migraine, depression, and anxiety. A model contract (a model file and a written specification), a Python reference implementation, and synthetic Fast Healthcare Interoperability Resources (FHIR) test patients define the expected outputs. One JavaScript scorer runs in a patient demonstration, a SMART on FHIR clinician application, and a CDS Hooks service. An atlas displays aggregate results. We evaluated the system on synthetic records with automated tests, cross-implementation output comparison, launches through the public SMART sandbox, and a counterexample review of the scoring and display rules. Results. The deployed Long COVID model had an area under the receiver operating characteristic curve of 0.67 in a random validation split. The eight self-report models reached 0.61 to 0.82 in identifying self-reporters without a code. The JavaScript scorer reproduced all 162 reference outputs exactly, and all 437 automated tests passed. A counterexample review found 5 classes of failure, including scoring ineligible patients and treating unavailable data as absent, all fixed and added as tests. Conclusions. Models fitted from released summary counts can be delivered through FHIR, SMART on FHIR, and CDS Hooks with reproducible scoring on synthetic records. Real-record interoperability, usability, and prospective clinical performance remain to be evaluated.

BackgroundClinicians increasingly use AI systems to search the medical literature, but current benchmarks do not jointly test whether a response identifies the originating paper and its supporting passage. LitBench evaluates four dimensions: stated versus interpretation-requiring facts, the number and composition of competing papers, one-versus two-paper evidence, and refusal when evidence is absent. MethodsLitBench comprises 1,980 open-access papers (1,000 epilepsy papers holding the answers, 980 decoys from unrelated fields) and 9,472 human-reviewed facts, giving 2,188 single-paper questions and 170 requiring a fact from each of two papers. Difficulty rose by burying the answer among up to 2,000 distractors, then again over live PubMed Central. The same questions were then asked with the answering paper removed, so that refusal was the correct response. Four systems were tested: Gemma-4B, Gemma-12B, Sonnet-5, and DeepSeek-V4-Flash (refusal conditions only). Three model judges scored each answer by majority; two-paper questions counted only when both facts were found. ResultsAcross fixed-corpus single-paper conditions, accuracy ranged from 55.9% to 71.9% for Gemma-4B, 61.2% to 75.5% for Gemma-12B, and 90.0% to 91.3% for Sonnet-5. Similar epilepsy papers were harder than mixed candidate sets for both Gemma configurations. Two-paper accuracy was 5.7%, 17.1%, and 37.8%, respectively. With evidence absent, Gemma-12B refusal fell from 88.3% to 39.9% as candidate sets grew; Gemma-4B almost never refused. Sonnet-5 refused on 94.9% of sampled single-paper and all sampled two-paper cells. DeepSeek-V4-Flash also refused frequently, but on 80.3% of matched answer-present controls. ConclusionsNo system was reliable across every axis tested: accuracy fell as competing papers were added, interpretation-requiring facts were harder than stated ones, two-paper synthesis was rarely achieved, and only some systems recognized absent evidence. What counts as "successful" literature search carries considerable nuance; LitBench can evaluate proposed tools at a fine-grained level.

Clinical records routinely document medical interventions but often omit the reasoning connecting patient-specific evidence to medical action. This limits the decision-relevant supervision available for training models for personalized care. Here we evaluate a data-augmentation approach that makes selected clinical decision relationships explicit in an 8-billion-parameter Qwen3 model, ChronIQ-DRx2. Using the same training cases and model architecture, expert-guided logic reconstruction was applied across four chronic diseases. On the held-out evaluation, the enhanced model had the highest plotted end-to-end task-completion rate among evaluated systems and higher plotted scores across diagnostic, medication and follow-up measures. The serious clinical error rates shown were 16.7% for the baseline and 12.5% for the enhanced model. In the human-AI evaluation, physician accuracy was 87.40% alone and 88.20% with AI assistance, while mean completion time was 0.959 and 0.780 min, respectively; medical-student accuracy was 62.71% and 74.80%, and mean completion time was 0.210 and 0.195 min. These results are descriptive point estimates; inferential comparisons are not reported here. Structured outputs may make recommendations easier to review, but rationale faithfulness, generalizability and effects on clinical safety require further evaluation.

Large language models (LLMs) are increasingly consulted for health advice, but they can produce fluent statements that are false or unsupported, known as hallucinations, and this risk is poorly characterised in low-resource languages such as Bangla. We measured hallucinations of locally deployable open-weight LLMs in Bangla diabetes question answering and tested whether retrieval-augmented generation (RAG) grounded in Bangladeshs National Guideline on Diabetes Mellitus reduces them. Forty patient questions spanning six clinical sub-domains were each answered twice, under zero-shot prompting, by Qwen3.5-9B, TigerLLM-9B, Qwen3.6-35B-A3B and Gemma-4-26B-A4B, and by BanglaDiaRAG, which adds hybrid dense (BGE-M3) and sparse (BM25) retrieval from a physician-reviewed Bangla guideline knowledge base to Qwen3.5-9B. Two physicians rated the 400 de-identified responses by consensus for medical accuracy, completeness, safety, readability and reliability on a 0-1 scale, and responses with accuracy below 0.70 were classified as hallucinated. Hallucination rates were 2.5% for Gemma-4-26B-A4B, 3.8% for TigerLLM-9B, 22.5% for Qwen3.6-35B-A3B and 77.5% for Qwen3.5-9B. Guideline grounding lowered the rate of Qwen3.5-9B to 11.3% (difference -66.3 percentage points; 95% CI -76.3 to -55.0) and raised every criterion by 0.24 to 0.38 points (Holm-adjusted p < 0.001). BanglaDiaRAG outscored Qwen3.6-35B-A3B on all five criteria and matched or exceeded TigerLLM-9B, but it remained below Gemma-4-26B-A4B in safety, readability and reliability, and its residual errors were specific, confident and potentially harmful. Guideline grounding can turn a weak general-purpose model into a much safer Bangla diabetes assistant, but the choice of base model mattered as much as retrieval and no configuration was free of hallucination, so clinical oversight remains essential before patient-facing use.

Background. Large language models (LLMs) are increasingly used to automate title-abstract screening in systematic reviews, but validation studies typically report sensitivity, specificity, or Cohen's kappa: metrics that are unstable under the low and unbalanced prevalence that characterises most reviews. No previous study has simultaneously combined a medical topic; a hybrid two-stage pipeline (rules + LLM); a three-way classification with a separately sampled uncertain class; stratified sampling with three independent human reviewers; and dual reporting of prevalence-robust coefficients (PABAK and Gwet's AC1). Objective. To validate a two-stage automated screening pipeline (a rule-based Python script followed by a reproducible LLM adjudication of uncertain records) against three independent human reviewers in a medical systematic review on high on-treatment platelet reactivity (HTPR), aspirin resistance, and carotid plaque. The primary validation target is the Stage 1 automatic-exclusion decision (670/904 records), which is the workload the pipeline is designed to take off the human reviewer. Methods. From 904 screened records (Stage 1: 131 INCLUDE, 670 EXCLUDE, 94 UNCERTAIN; Stage 2 LLM via API, temperature = 0: 11 INCLUDE, 83 EXCLUDE), a stratified sample of 185 records (50 per Stage 1 stratum + 35 Stage 2; seed = 42) was judged independently and blind to the automated labels by three physicians: E.P. (pipeline designer), D.C. and P.P.B. (both external to the thesis project). Concordance was quantified with sensitivity, specificity, PABAK (q = 3 throughout), Gwet's AC1, work saved over sampling (WSS), and number needed to read (NNR), AI versus each reviewer and between reviewers, overall and by stratum. Majority labels (2 of 3) are reported as a supplementary panel; they do not replace pairwise tables as the primary reference for sensitivity. Results. On the primary target, Stage1-EXCLUDE concordance was PABAK = AC1 = 1.000 versus E.P. (48/48) and versus D.C. (50/50), and PABAK = 0.940 / AC1 = 0.959 versus Berti (48/50 EXCLUDE, 2 UNCERTAIN). That stratum is 670/904 records (74.1%). Pairwise human agreement was highest between the two external readers (Costazza vs Berti: PABAK = 0.854, AC1 = 0.881) and lower when E.P. was one of the pair (0.703 / 0.774 vs Costazza; 0.621 / 0.702 vs Berti). AI versus each human remained poor (PABAK 0.225 / 0.205 / 0.222). Majority labels on 185 records were 146 EXCLUDE / 27 UNCERTAIN / 4 INCLUDE / 8 NO_MAJORITY. The machine labelled INCLUDE 0 of the 4 majority-INCLUDE records (74, 75, 76, 80). Overall sensitivity was 0.800 versus E.P. (8/10; false negatives 623, 712), 0.500 versus D.C. (1/2; false negative 76), and 0.833 versus Berti (5/6; false negative 76). Versus majority the false-negative count is 1 (record 76); records 623 and 712 are majority EXCLUDE. Stage1-INCLUDE sensitivity of 1.000 is tautological. WSS = 74.1% describes the exclusion stratum only. Conclusions. What this study validates, now with three independent physicians, is automatic exclusion at Stage 1. Fine classification of INCLUDE records is not validated: humans do not agree on which residual records are INCLUDE (triple INCLUDE intersection = record 74 only), Stage 2 missed most of the few human INCLUDE labels it received, and the machine recovered none of the four majority-INCLUDE records. Domain-familiarity as an explanation of INCLUDE divergence remains a hypothesis, not a measured variable.